Training-related changes in neural beta oscillations associated with implicit and explicit motor sequence learning.

Many motor actions we perform have a sequential nature while learning a motor sequence involves both implicit and explicit processes. In this work, we developed a task design where participants concurrently learn an implicit and an explicit motor sequence across five training sessions, with EEG recordings at sessions 1 and 5. This intra-subject approach allowed us to study training-induced behavioral and neural changes specific to the explicit and implicit components. Based on previous reports of beta power modulations in sensorimotor networks related to sequence learning, we focused our analysis on beta oscillations at motor-cortical sites. On a behavioral level, substantial performance gains were evident early in learning in the explicit condition, plus slower performance gains across training sessions in both explicit and implicit sequence learning. Consistent with the behavioral trends, we observed a training-related increase in beta power in both sequence learning conditions, while the explicit condition displayed stronger beta power suppression during early learning. The initially stronger beta suppression and subsequent increase in beta power specific to the explicit component, correlated with enhanced behavioral performance, possibly reflecting higher cortical excitability. Our study suggests an involvement of motor-cortical beta oscillations in the explicit component of motor sequence learning.

ConTraNet: A hybrid network for improving the classification of EEG and EMG signals with limited training data.

OBJECTIVE: Bio-Signals such as electroencephalography (EEG) and electromyography (EMG) are widely used for the rehabilitation of physically disabled people and for the characterization of cognitive impairments. Successful decoding of these bio-signals is however non-trivial because of the time-varying and non-stationary characteristics. Furthermore, existence of short- and long-range dependencies in these time-series signal makes the decoding even more challenging. State-of-the-art studies proposed Convolutional Neural Networks (CNNs) based architectures for the classification of these bio-signals, which are proven useful to learn spatial representations. However, CNNs because of the fixed size convolutional kernels and shared weights pay only uniform attention and are also suboptimal in learning short-long term dependencies, simultaneously, which could be pivotal in decoding EEG and EMG signals. Therefore, it is important to address these limitations of CNNs. To learn short- and long-range dependencies simultaneously and to pay more attention to more relevant part of the input signal, Transformer neural network-based architectures can play a significant role. Nonetheless, it requires a large corpus of training data. However, EEG and EMG decoding studies produce limited amount of the data. Therefore, using standalone transformers neural networks produce ordinary results. In this study, we ask a question whether we can fix the limitations of CNN and transformer neural networks and provide a robust and generalized model that can simultaneously learn spatial patterns, long-short term dependencies, pay variable amount of attention to time-varying non-stationary input signal with limited training data. APPROACH: In this work, we introduce a novel single hybrid model called ConTraNet, which is based on CNN and Transformer architectures that contains the strengths of both CNN and Transformer neural networks. ConTraNet uses a CNN block to introduce inductive bias in the model and learn local dependencies, whereas the Transformer block uses the self-attention mechanism to learn the short- and long-range or global dependencies in the signal and learn to pay different attention to different parts of the signals. MAIN RESULTS: We evaluated and compared the ConTraNet with state-of-the-art methods on four publicly available datasets (BCI Competition IV dataset 2b, Physionet MI-EEG dataset, Mendeley sEMG dataset, Mendeley sEMG V1 dataset) which belong to EEG-HMI and EMG-HMI paradigms. ConTraNet outperformed its counterparts in all the different category tasks (2-class, 3-class, 4-class, 7-class, and 10-class decoding tasks). SIGNIFICANCE: With limited training data ConTraNet significantly improves classification performance on four publicly available datasets for 2, 3, 4, 7, and 10-classes compared to its counterparts.

Phantom touch illusion, an unexpected phenomenological effect of tactile gating in the absence of tactile stimulation.

We report the presence of a tingling sensation perceived during self-touch without physical stimulation. We used immersive virtual reality scenarios in which subjects touched their body using a virtual object. This touch resulted in a tingling sensation corresponding to the location touched on the virtual body. We called it "phantom touch illusion" (PTI). Interestingly, the illusion was also reported when subjects touched invisible (inferred) parts of their limb. We reason that this PTI results from tactile gating process during self-touch if there is no tactile input to supress. The reported PTI when touching invisible body parts indicates that tactile gating is not exclusively based on vision, but rather on multi-sensory, top-down input involving body schema. This supplementary finding shows that representations of one's own body are defined top-down, beyond the available sensory information.

Neural representations of economic decision variables in human posterior parietal cortex.

Decision making has been intensively studied in the posterior parietal cortex in non-human primates on a single neuron level. In humans decision making has mainly been studied with psychophysical tools or with fMRI. Here, we investigated how single neurons from human posterior parietal cortex represent numeric values informing future decisions during a complex two-player game. The tetraplegic study participant was implanted with a Utah electrode array in the anterior intraparietal area (AIP). We played a simplified variant of Black Jack with the participant while neuronal data was recorded. During the game two players are presented with numbers which are added up. Each time a number is presented the player has to decide to proceed or to stop. Once the first player stops or the score reaches a limit the turn passes on to the second player who tries to beat the score of the first player. Whoever is closer to the limit (without overshooting) wins the game. We found that many AIP neurons selectively responded to the face value of the presented number. Other neurons tracked the cumulative score or were selectively active for the upcoming decision of the study participant. Interestingly, some cells also kept track of the opponent's score. Our findings show that parietal regions engaged in hand action control also represent numbers and their complex transformations. This is also the first demonstration of complex economic decisions being possible to track in single neuron activity in human AIP. Our findings show how tight are the links between parietal neural circuits underlying hand control, numerical cognition and complex decision-making.

Error-Related Potentials in Reinforcement Learning-Based Brain-Machine Interfaces.

The human brain has been an object of extensive investigation in different fields. While several studies have focused on understanding the neural correlates of error processing, advances in brain-machine interface systems using non-invasive techniques further enabled the use of the measured signals in different applications. The possibility of detecting these error-related potentials (ErrPs) under different experimental setups on a single-trial basis has further increased interest in their integration in closed-loop settings to improve system performance, for example, by performing error correction. Fewer works have, however, aimed at reducing future mistakes or learning. We present a review focused on the current literature using non-invasive systems that have combined the ErrPs information specifically in a reinforcement learning framework to go beyond error correction and have used these signals for learning.

Enhancing the decoding accuracy of EEG signals by the introduction of anchored-STFT and adversarial data augmentation method.

Brain-computer interfaces (BCIs) enable communication between humans and machines by translating brain activity into control commands. Electroencephalography (EEG) signals are one of the most used brain signals in non-invasive BCI applications but are often contaminated with noise. Therefore, it is possible that meaningful patterns for classifying EEG signals are deeply hidden. State-of-the-art deep-learning algorithms are successful in learning hidden, meaningful patterns. However, the quality and the quantity of the presented inputs are pivotal. Here, we propose a feature extraction method called anchored Short Time Fourier Transform (anchored-STFT), which is an advanced version of STFT, as it minimizes the trade-off between temporal and spectral resolution presented by STFT. In addition, we propose a data augmentation method derived from l2-norm fast gradient sign method (FGSM), called gradient norm adversarial augmentation (GNAA). GNAA is not only an augmentation method but is also used to harness adversarial inputs in EEG data, which not only improves the classification accuracy but also enhances the robustness of the classifier. In addition, we also propose a CNN architecture, namely Skip-Net, for the classification of EEG signals. The proposed pipeline outperforms the current state-of-the-art methods and yields classification accuracies of 90.7% on BCI competition II dataset III and 89.5%, 81.8%, 76.0% and 85.4%, 69.1%, 80.9% on different data distributions of BCI Competition IV dataset 2b and 2a, respectively.

Data-efficient Causal Decoding of Spiking Neural Activity using Weighted Voting.

Brain-Computer Interface systems can contribute to a vast set of applications such as overcoming physical disabilities in people with neural injuries or hands-free control of devices in healthy individuals. However, having systems that can accurately interpret intention online remains a challenge in this field. Robust and data-efficient decoding-despite the dynamical nature of cortical activity and causality requirements for physical function-is among the most important challenges that limit the widespread use of these devices for real-world applications. Here, we present a causal, data-efficient neural decoding pipeline that predicts intention by first classifying recordings in short sliding windows. Next, it performs weighted voting over initial predictions up to the current point in time to report a refined final prediction. We demonstrate its utility by classifying spiking neural activity collected from the human posterior parietal cortex for a cue, delay, imaginary motor task. This pipeline provides higher classification accuracy than state-of-the-art time windowed spiking activity based causal methods, and is robust to the choice of hyper-parameters.

Quantifying the alignment error and the effect of incomplete somatosensory feedback on motor performance in a virtual brain-computer-interface setup.

Invasive brain-computer-interfaces (BCIs) aim to improve severely paralyzed patient's (e.g. tetraplegics) quality of life by using decoded movement intentions to let them interact with robotic limbs. We argue that the performance in controlling an end-effector using a BCI depends on three major factors: decoding error, missing somatosensory feedback and alignment error caused by translation and/or rotation of the end-effector relative to the real or perceived body. Using a virtual reality (VR) model of an ideal BCI decoder with healthy participants, we found that a significant performance loss might be attributed solely to the alignment error. We used a shape-drawing task to investigate and quantify the effects of robot arm misalignment on motor performance independent from the other error sources. We found that a 90° rotation of the robot arm relative to the participant leads to the worst performance, while we did not find a significant difference between a 45° rotation and no rotation. Additionally, we compared a group of subjects with indirect haptic feedback with a group without indirect haptic feedback to investigate the feedback-error. In the group without feedback, we found a significant difference in performance only when no rotation was applied to the robot arm, supporting that a form of haptic feedback is another important factor to be considered in BCI control.

SpikeDeep-classifier: a deep-learning based fully automatic offline spike sorting algorithm.

Objective.Advancements in electrode design have resulted in micro-electrode arrays with hundreds of channels for single cell recordings. In the resulting electrophysiological recordings, each implanted electrode can record spike activity (SA) of one or more neurons along with background activity (BA). The aim of this study is to isolate SA of each neural source. This process is called spike sorting or spike classification. Advanced spike sorting algorithms are time consuming because of the human intervention at various stages of the pipeline. Current approaches lack generalization because the values of hyperparameters are not fixed, even for multiple recording sessions of the same subject. In this study, a fully automatic spike sorting algorithm called 'SpikeDeep-Classifier' is proposed. The values of hyperparameters remain fixed for all the evaluation data.Approach.The proposed approach is based on our previous study (SpikeDeeptector) and a novel background activity rejector (BAR), which are both supervised learning algorithms and an unsupervised learning algorithm (K-means). SpikeDeeptector and BAR are used to extract meaningful channels and remove BA from the extracted meaningful channels, respectively. The process of clustering becomes straight-forward once the BA is completely removed from the data. Then, K-means with a predefined maximum number of clusters is applied on the remaining data originating from neural sources only. Lastly, a similarity-based criterion and a threshold are used to keep distinct clusters and merge similar looking clusters. The proposed approach is called cluster accept or merge (CAOM) and it has only two hyperparameters (maximum number of clusters and similarity threshold) which are kept fixed for all the evaluation data after tuning.Main results.We compared the results of our algorithm with ground-truth labels. The algorithm is evaluated on data of human patients and publicly available labeled non-human primates (NHPs) datasets. The average accuracy of BAR on datasets of human patients is 92.3% which is further reduced to 88.03% after (K-means + CAOM). In addition, the average accuracy of BAR on a publicly available labeled dataset of NHPs is 95.40% which reduces to 86.95% after (K-mean + CAOM). Lastly, we compared the performance of the SpikeDeep-Classifier with two human experts, where SpikeDeep-Classifier has produced comparable results.Significance.The SpikeDeep-Classifier is evaluated on the datasets of multiple recording sessions of different species, different brain areas and different electrode types without further retraining. The results demonstrate that 'SpikeDeep-Classifier' possesses the ability to generalize well on a versatile dataset and henceforth provides a generalized and fully automated solution to offline spike sorting.Clinical trial registration numberThe clinical trial registration number for patients implanted with the Utah array isNCT 01849822.For the epilepsy patients, approval from the local ethics committee at the Ruhr-University Bochum, Germany, was obtained prior to implantation. The Clinical trial registration number for the epilepsy patients implanted with microwires is16-5670.

SpikeDeeptector: a deep-learning based method for detection of neural spiking activity.

OBJECTIVE: In electrophysiology, microelectrodes are the primary source for recording neural data (single unit activity). These microelectrodes can be implanted individually or in the form of arrays containing dozens to hundreds of channels. Recordings of some channels contain neural activity, which are often contaminated with noise. Another fraction of channels does not record any neural data, but only noise. By noise, we mean physiological activities unrelated to spiking, including technical artifacts and neural activities of neurons that are too far away from the electrode to be usefully processed. For further analysis, an automatic identification and continuous tracking of channels containing neural data is of great significance for many applications, e.g. automated selection of neural channels during online and offline spike sorting. Automated spike detection and sorting is also critical for online decoding in brain-computer interface (BCI) applications, in which only simple threshold crossing events are often considered for feature extraction. To our knowledge, there is no method that can universally and automatically identify channels containing neural data. In this study, we aim to identify and track channels containing neural data from implanted electrodes, automatically and more importantly universally. By universally, we mean across different recording technologies, different subjects and different brain areas. APPROACH: We propose a novel algorithm based on a new way of feature vector extraction and a deep learning method, which we call SpikeDeeptector. SpikeDeeptector considers a batch of waveforms to construct a single feature vector and enables contextual learning. The feature vectors are then fed to a deep learning method, which learns contextualized, temporal and spatial patterns, and classifies them as channels containing neural spike data or only noise. MAIN RESULTS: We trained the model of SpikeDeeptector on data recorded from a single tetraplegic patient with two Utah arrays implanted in different areas of the brain. The trained model was then evaluated on data collected from six epileptic patients implanted with depth electrodes, unseen data from the tetraplegic patient and data from another tetraplegic patient implanted with two Utah arrays. The cumulative evaluation accuracy was 97.20% on 1.56 million hand labeled test inputs. SIGNIFICANCE: The results demonstrate that SpikeDeeptector generalizes not only to the new data, but also to different brain areas, subjects, and electrode types not used for training. CLINICAL TRIAL REGISTRATION NUMBER: The clinical trial registration number for patients implanted with the Utah array is NCT01849822. For the epilepsy patients, approval from the local ethics committee at the Ruhr-University Bochum, Germany, was obtained prior to implantation.

Engineering Artificial Somatosensation Through Cortical Stimulation in Humans.

Sensory feedback is a critical aspect of motor control rehabilitation following paralysis or amputation. Current human studies have demonstrated the ability to deliver some of this sensory information via brain-machine interfaces, although further testing is needed to understand the stimulation parameters effect on sensation. Here, we report a systematic evaluation of somatosensory restoration in humans, using cortical stimulation with subdural mini-electrocorticography (mini-ECoG) grids. Nine epilepsy patients undergoing implantation of cortical electrodes for seizure localization were also implanted with a subdural 64-channel mini-ECoG grid over the hand area of the primary somatosensory cortex (S1). We mapped the somatotopic location and size of receptive fields evoked by stimulation of individual channels of the mini-ECoG grid. We determined the effects on perception by varying stimulus parameters of pulse width, current amplitude, and frequency. Finally, a target localization task was used to demonstrate the use of artificial sensation in a behavioral task. We found a replicable somatotopic representation of the hand on the mini-ECoG grid across most subjects during electrical stimulation. The stimulus-evoked sensations were usually of artificial quality, but in some cases were more natural and of a cutaneous or proprioceptive nature. Increases in pulse width, current strength and frequency generally produced similar quality sensations at the same somatotopic location, but with a perception of increased intensity. The subjects produced near perfect performance when using the evoked sensory information in target acquisition tasks. These findings indicate that electrical stimulation of somatosensory cortex through mini-ECoG grids has considerable potential for restoring useful sensation to patients with paralysis and amputation.

Hand Shape Representations in the Human Posterior Parietal Cortex.

Humans shape their hands to grasp, manipulate objects, and to communicate. From nonhuman primate studies, we know that visual and motor properties for grasps can be derived from cells in the posterior parietal cortex (PPC). Are non-grasp-related hand shapes in humans represented similarly? Here we show for the first time how single neurons in the PPC of humans are selective for particular imagined hand shapes independent of graspable objects. We find that motor imagery to shape the hand can be successfully decoded from the PPC by implementing a version of the popular Rock-Paper-Scissors game and its extension Rock-Paper-Scissors-Lizard-Spock. By simultaneous presentation of visual and auditory cues, we can discriminate motor imagery from visual information and show differences in auditory and visual information processing in the PPC. These results also demonstrate that neural signals from human PPC can be used to drive a dexterous cortical neuroprosthesis. SIGNIFICANCE STATEMENT: This study shows for the first time hand-shape decoding from human PPC. Unlike nonhuman primate studies in which the visual stimuli are the objects to be grasped, the visually cued hand shapes that we use are independent of the stimuli. Furthermore, we can show that distinct neuronal populations are activated for the visual cue and the imagined hand shape. Additionally we found that auditory and visual stimuli that cue the same hand shape are processed differently in PPC. Early on in a trial, only the visual stimuli and not the auditory stimuli can be decoded. During the later stages of a trial, the motor imagery for a particular hand shape can be decoded for both modalities.

Neurophysiology. Decoding motor imagery from the posterior parietal cortex of a tetraplegic human.

Nonhuman primate and human studies have suggested that populations of neurons in the posterior parietal cortex (PPC) may represent high-level aspects of action planning that can be used to control external devices as part of a brain-machine interface. However, there is no direct neuron-recording evidence that human PPC is involved in action planning, and the suitability of these signals for neuroprosthetic control has not been tested. We recorded neural population activity with arrays of microelectrodes implanted in the PPC of a tetraplegic subject. Motor imagery could be decoded from these neural populations, including imagined goals, trajectories, and types of movement. These findings indicate that the PPC of humans represents high-level, cognitive aspects of action and that the PPC can be a rich source for cognitive control signals for neural prosthetics that assist paralyzed patients.

A cognitive neuroprosthetic that uses cortical stimulation for somatosensory feedback.

OBJECTIVE: Present day cortical brain-machine interfaces (BMIs) have made impressive advances using decoded brain signals to control extracorporeal devices. Although BMIs are used in a closed-loop fashion, sensory feedback typically is visual only. However medical case studies have shown that the loss of somesthesis in a limb greatly reduces the agility of the limb even when visual feedback is available. APPROACH: To overcome this limitation, this study tested a closed-loop BMI that utilizes intracortical microstimulation to provide 'tactile' sensation to a non-human primate. MAIN RESULT: Using stimulation electrodes in Brodmann area 1 of somatosensory cortex (BA1) and recording electrodes in the anterior intraparietal area, the parietal reach region and dorsal area 5 (area 5d), it was found that this form of feedback can be used in BMI tasks. SIGNIFICANCE: Providing somatosensory feedback has the poyential to greatly improve the performance of cognitive neuroprostheses especially for fine control and object manipulation. Adding stimulation to a BMI system could therefore improve the quality of life for severely paralyzed patients.

Toward more versatile and intuitive cortical brain-machine interfaces.

Brain-machine interfaces have great potential for the development of neuroprosthetic applications to assist patients suffering from brain injury or neurodegenerative disease. One type of brain-machine interface is a cortical motor prosthetic, which is used to assist paralyzed subjects. Motor prosthetics to date have typically used the motor cortex as a source of neural signals for controlling external devices. The review will focus on several new topics in the arena of cortical prosthetics. These include using: recordings from cortical areas outside motor cortex; local field potentials as a source of recorded signals; somatosensory feedback for more dexterous control of robotics; and new decoding methods that work in concert to form an ecology of decode algorithms. These new advances promise to greatly accelerate the applicability and ease of operation of motor prosthetics.

Sensorimotor learning biases choice behavior: a learning neural field model for decision making.

According to a prominent view of sensorimotor processing in primates, selection and specification of possible actions are not sequential operations. Rather, a decision for an action emerges from competition between different movement plans, which are specified and selected in parallel. For action choices which are based on ambiguous sensory input, the frontoparietal sensorimotor areas are considered part of the common underlying neural substrate for selection and specification of action. These areas have been shown capable of encoding alternative spatial motor goals in parallel during movement planning, and show signatures of competitive value-based selection among these goals. Since the same network is also involved in learning sensorimotor associations, competitive action selection (decision making) should not only be driven by the sensory evidence and expected reward in favor of either action, but also by the subject's learning history of different sensorimotor associations. Previous computational models of competitive neural decision making used predefined associations between sensory input and corresponding motor output. Such hard-wiring does not allow modeling of how decisions are influenced by sensorimotor learning or by changing reward contingencies. We present a dynamic neural field model which learns arbitrary sensorimotor associations with a reward-driven Hebbian learning algorithm. We show that the model accurately simulates the dynamics of action selection with different reward contingencies, as observed in monkey cortical recordings, and that it correctly predicted the pattern of choice errors in a control experiment. With our adaptive model we demonstrate how network plasticity, which is required for association learning and adaptation to new reward contingencies, can influence choice behavior. The field model provides an integrated and dynamic account for the operations of sensorimotor integration, working memory and action selection required for decision making in ambiguous choice situations.

Choosing goals, not rules: deciding among rule-based action plans.

In natural situations, movements are often directed toward locations different from that of the evoking sensory stimulus. Movement goals must then be inferred from the sensory cue based on rules. When there is uncertainty about the rule that applies for a given cue, planning a movement involves both choosing the relevant rule and computing the movement goal based on that rule. Under these conditions, it is not clear whether primates compute multiple movement goals based on all possible rules before choosing an action, or whether they first choose a rule and then only represent the movement goal associated with that rule. Supporting the former hypothesis, we show that neurons in the frontoparietal reach areas of monkeys simultaneously represent two different rule-based movement goals, which are biased by the monkeys' choice preferences. Apparently, primates choose between multiple behavioral options by weighing against each other the movement goals associated with each option.

The cortical timeline for deciding on reach motor goals.

Flexible sensorimotor planning is the basis for goal-directed behavior. We investigated the integration of visuospatial information with context-specific transformation rules during reach planning. We were especially interested in the relative timing of motor-goal decisions in monkey dorsal premotor cortex (PMd) and parietal reach region (PRR). We used a rule-based mapping task with different cueing conditions to compare task-dependent motor-goal latencies. The task allowed us a separation of cue-related from motor-related activity, and a separation of activity related to motor planning from activity related to motor initiation or execution. The results show that selectivity for the visuospatial goal of a pending movement occurred earlier in PMd than PRR whenever the task required spatial remapping. Such remapping was needed if the spatial representation of a cue or of a default motor plan had to be transformed into a spatially incongruent representation of the motor goal. In contrast, we did not find frontoparietal latency differences if the spatial representation of the cue or the default plan was spatially congruent with the motor goal. The fact that frontoparietal latency differences occurred only in conditions with spatial remapping was independent of the subjects' partial a priori knowledge about the pending goal. Importantly, frontoparietal latency differences existed for motor-goal representations during movement planning, without immediate motor execution. We interpret our findings as being in support of the hypothesis that latency differences reflect a dynamic reorganization of network activity in PRR, and suggest that the reorganization is contingent on frontoparietal projections from PMd.

Implementation of spatial transformation rules for goal-directed reaching via gain modulation in monkey parietal and premotor cortex.

Planning goal-directed movements requires the combination of visuospatial with abstract contextual information. Our sensory environment constrains possible movements to a certain extent. However, contextual information guides proper choice of action in a given situation and allows flexible mapping of sensory instruction cues onto different motor actions. We used anti-reach tasks to test the hypothesis that spatial motor-goal representations in cortical sensorimotor areas are gain modulated by the behavioral context to achieve flexible remapping of spatial cue information onto arbitrary motor goals. We found that gain modulation of neuronal reach goal representations is commonly induced by the behavioral context in individual neurons of both, the parietal reach region (PRR) and the dorsal premotor cortex (PMd). In addition, PRR showed stronger directional selectivity during the planning of a reach toward a directly cued goal (pro-reach) compared with an inferred target (anti-reach). PMd, however, showed stronger overall activity during reaches toward inferred targets compared with directly cued targets. Based on our experimental evidence, we suggest that gain modulation is the computational mechanism underlying the integration of spatial and contextual information for flexible, rule-driven stimulus-response mapping, and thereby forms an important basis of goal-directed behavior. Complementary contextual effects in PRR versus PMd are consistent with the idea that posterior parietal cortex preferentially represents sensory-driven, "automatic" motor goals, whereas frontal sensorimotor areas are stronger engaged in the representation of rule-based, "inferred" motor goals.